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DETAILED ACTION 

1. Claims 33-47 and 49-61 are pending. 

2. The rejection to claims 33-38, 40-47 and 49-61 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over McGee et al. (US 4942539 A) in view of Wei et al. (Multisensory Visual 
servoing by a Neural Network, 1999) is withdrawn. 

3. The objection to claim 39 is withdrawn. 

4. Applicant's arguments with respect to claims 33-47 and 49-61 have been considered but 
are moot in view of the new ground(s) of rejection. See rejection below. 

Claim Rejections - 35 USC § 112 

5. The following is a quotation of the first paragraph of 35 U.S.C. 1 12: 

The specification shall contain a written description of the invention, and of the manner and process of making 
and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it 
pertains, or with which it is most nearly connected, to make and use the same and shall set forth the best mode 
contemplated by the inventor of carrying out his invention. 

6. Claim 50 is rejected under 35 U.S.C. 1 12, first paragraph, as failing to comply with the 
written description requirement. The claim(s) contains subject matter which was not described 
in the specification in such a way as to reasonably convey to one skilled in the relevant art that 
the inventor(s), at the time the application was filed, had possession of the claimed invention. 
The phrase "using only the single captured image" does not have any support in the 
specification. 
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Specification 



7. The amendment filed 06/05/2009 is objected to under 35 U.S.C. 132(a) because it 
introduces new matter into the disclosure. 35 U.S.C. 132(a) states that no amendment shall 
introduce new matter into the disclosure of the invention. The added material which is not 
supported by the original disclosure is as follows: See claim 50, wherein the phrase "using only 
the single captured image" does not have any support in the specification. Applicant is required 
to cancel the new matter in the reply to this Office Action. 



8. The abstract of the disclosure is objected to because the web address should not be in the 
specification on page 6, line 11, "http://www.ai.mit.edu/people/bkph/papers/tsaiexplain.pdf. 
Correction is required. See MPEP § 608.01(b). 



Claim Rejections - 35 USC § 103 



9. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

10. This application currently names joint inventors. In considering patentability of the 
claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various 
claims was commonly owned at the time any inventions covered therein were made absent any 
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evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out 
the inventor and invention dates of each claim that was not commonly owned at the time a later 
invention was made in order for the examiner to consider the applicability of 35 U.S.C. 103(c) 
and potential 35 U.S.C. 102(e), (f) or (g) prior art under 35 U.S.C. 103(a). 

11. Claims 33-35, 38, 41, 42 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Matusik et al. (US 20020159628) in view of McGee et al. (4,942,539). 

A per claim 33, Matusik et al. (US 20020159628) teaches capturing a number of images 
(see abs. particularly "A system digitizes a three-dimensional object as a three-dimension model 
by ") of a calibration object by the camera (see section [0022]); 

determining a set of intrinsic parameters of the camera (see sections [0027-0028]) from at least 
one of the number of images (see abs. particularly "A system digitizes a three-dimensional object 
as a three- dimension model by ") of the calibration object captured by the camera (see section 
[0026]); and determining a set of extrinsic parameters of the camera from at least one of the 
number of images (see abs. particularly "A system digitizes a three-dimensional object as a 
three-dimension model by") of the calibration object captured by the camera (see section 
[0026]), the set of extrinsic parameters comprising (see sections [0027 and 0028]). Matusik et al. 
does not specifically teach a camera space-to-training space transformation defining a 
transformation between a camera space reference frame and a training space reference frame. 

McGee et al. (4,942,539) teaches a camera space-to-training space transformation 
defining a transformation between a camera space reference frame and a training space reference 
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frame (see abs., wherein reference data has been reference frame; and "The offset is then 
transformed into the coordinate system or frame" has been taken defined transformation; and fig. 
1, has shown evidence of a camera placed above the object which has been considered as space- 
to-training). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., with the teaching of McGee et al, 
because this modification would have introduced frame into frame into Matusik's et al. teaching, 
thereby "The camera is calibrated to determine internal camera geometric and optical 
characteristics (intrinsic parameters) and the 3-D position and orientation of the camera frame 
relative to the coordinate system at the station 12." (see McGee et al. col. 5, lines 54-58). 

A per claim 34, Matusik et al. teaches a method positioning the camera with respect to 
the calibration object (see sections [0023 and 0026]). 

A per claim 35, Matusik et al. teaches method wherein positioning the camera with 
respect to the calibration object (see sections [0023 and 0026]) comprises positioning the camera 
orthogonally with respect to a ruled template with a number of features (see fig. 1, wherein 
element 130 being placed orthogonally), where a known or determinable physical relationship 
exists between at least some of the features (see fig. 2, wherein physical relationship exists 
between element 130 and the object 150). 
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A per claim 38, Matusik et al. teaches a wherein capturing a number of images (see abs. 
particularly "A system digitizes a three-dimensional object as a three- dimension model by ") of 
calibration object by the camera (see section [0022]) comprises capturing at least one image at 
each of a plurality of different orientations with respect to the calibration object (see section 
[0025]). 



A per claim 41, Matusik et al. teaches a further comprising: based at least in part on at 
least two of the number of images (see abs. particularly "A system digitizes a three-dimensional 
object as a three-dimension model" and fig. 1 , element 130, wherein each has been considered 
image taken means) captured by the camera of the calibration object (see section [0026]). 
Matusik et al. does not specifically teach determining a camera space-to-tool space 
transformation. 

McGee et al. (4,942,539) teaches a camera space-to-tool space transformation (see abs., 
wherein reference data has been reference frame; and "The offset is then transformed into the 
coordinate system or frame" has been taken defined transformation; and fig. 1, has a camera 
which has been considered as space-to-tool). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., with the teaching of McGee et al., 
because this modification would have introduced frame into frame into Matusik's et al. teaching 
thereby "The camera is calibrated to determine internal camera geometric and optical 
characteristics (intrinsic parameters) and the 3-D position and orientation of the camera frame 
relative to the coordinate system at the station 12." (see McGee's et al. col. 5, lines 54-58). 
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A per claim 42, Matusik et al. teaches a further comprising: determining a camera space- 
to-tool space transformation based on single one of the number of images (see abs. particularly 
"A system digitizes a three-dimensional object as a three-dimension model") captured by the 
camera of the calibration object (see section [0026]). Matusik et al. does not specifically teach a 
number of physical coordinates of at least one feature of the calibration object. 

McGee et al. (4,942,539) teaches a number of physical coordinates of at least one feature 
of the calibration object (see abs., and fig. 1, has a object which has been as having physical 
coordinates). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., with the teaching of McGee et al., 
because this modification would have introduced coordinate into frame into Matusik' s et al. 
teaching, thereby "The camera is calibrated to determine internal camera geometric and optical 
characteristics (intrinsic parameters) and the 3-D position and orientation of the camera frame 
relative to the coordinate system at the station 12." (see McGee's et al. col. 5, lines 54-58). 

12. Claim 36, 37 are rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik et 
al. in view of McGee et al. as applied to claim 33 above, and further in view of Nayar et al. (US 
5,802,201). 

A per claim 36, Matusik et al. teaches a method wherein positioning the camera with 
respect to the calibration object (see sections [0023 and 0026]) comprises positioning the camera 
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with respect to a sample of a type of object (see fig. 1, elements 130 and 150, and element 150 
has been taken for the object sample) where a known or determinable physical relationship exists 
between at least some of the features (see fig. 2, wherein physical relationship exists between 
element 130 and the object 150). Matusik et al. does not specifically teach a robot that that will 
manipulate, the sample having a number of features. 

Nayar et al. (US 5,802,201) teaches a robot (see fig. 2, element 100) that will manipulate, 
the sample having a number of features (see col. 5, lines 23-35). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., with the teaching of Nayar et al., 
because this modification would have (introduced, provided, increased, added, enhanced) into 
Matusik's et al. teaching, thereby "" (see Nayar's et al.). 

A per claim 37, Matusik et al. teaches a method wherein capturing a number of images 
(see abs. particularly "A system digitizes a three-dimensional object as a three-dimension model 
by ") of a calibration object by the camera (see section [0022]) comprises capturing at least one 
image at each of a plurality of positions (see section [0024]). Matusik et al. does not specifically 
teach capturing image spaced perpendicularly from the calibration object. 

Nayar et al. teaches a robot capturing image spaced perpendicularly from the calibration 
object (see abs. and fig. 1, wherein the camera has been place on top of the object 
perpendicularly). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., with the teaching of Nayar et al., 
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because this modification would have (introduced, provided, increased, added, enhanced) into 
Matusik's et al. teaching, thereby "" (see Nayar's et al.). 

13. Claim 39 are rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik et al. in 
view of McGee et al. as applied to claim 33 above, and further in view of Armstrong et al. (US 
20040233461) and Rhoads (US 20100040255).. 

A per claim 39, Matusik et al. teaches in combination with McGee et al. a wherein 
determining a set of intrinsic parameters of the camera (see sections [0027-0028]) from the at 
least one of the number of images (see abs. particularly "A system digitizes a three-dimensional 
object as a three-dimension model by ") of the calibration object captured by the camera (see 
section [0026]). Matusik et al. does not specifically teach determining at least one of a focal 
length, a first order radial lens distortion coefficient, a set of coordinates of a center of a radial 
lens distortion, or a scale factor indicative of a frame grabber scanline resampling uncertainty. 

Armstrong et al. (US 20040233461) teaches determining at least one of a focal length, a 
first order radial lens distortion coefficient (see sections [0054 and 0055], wherein the radial 
distortion has been considered as first order coefficient), a set of coordinates of a center of a 
radial lens distortion (see section [0056]), or a scale factor (see sections [0011, 0053 and 0083]) 
indicative of a framegrabber scanline resampling uncertainty (see sections [0374 and 0379], for 
scaling). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al. and McGee et al., with the teaching 
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of Armstrong et al. because this modification would have introduced translation and rotation 
into Matusik's et al. and McGee's et al. teaching, thereby "camera coordinate systems may be 
rotated in three-dimensional space with respect to the other" (see Armstrong's et al. section 
[0026]). 

Although, Armstrong et al. in combination of Matusik et al. teach a scale, but not a scale 
factor, and framegrabber. 

Rhoads (US 20100040255) teaches the well known features of scale factor (see section 
[0165]), and framegrabber (see section [0306]). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., McGee et al, and Armstrong et al. 
with the teaching of Rhoads, because this modification would have introduced scale factor into 
Matusik's et al. and McGee's et al. teaching, thereby "processing data representing video" (see 
Rhoads' abs.). 

14. Claim 40 are rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik et al. in 
view of McGee et al. as applied to claim 33 above, and further in view of Armstrong et al. (US 
20040233461).. 

A per claim 40, Matusik et al. teaches a wherein determining a set of extrinsic parameters 
of the camera from at least one of the number of images (see abs. particularly "A system 
digitizes a three-dimensional object as a three-dimension model by ") of the calibration object 
captured by the camera (see section [0026]), the set of extrinsic parameters comprising (see 
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sections [0027 and 0028]). Matusik et al. does not specifically teach a camera space-to- training 
space transformation defining a transformation between a camera space reference frame and a 
training space reference frame. 

McGee et al. (4,942,539) teaches a camera space-to-training space transformation 
defining a transformation between a camera space reference frame and a training space reference 
frame (see abs., wherein reference data has been reference frame; and "The offset is then 
transformed into the coordinate system or frame" has been taken defined transformation; and fig. 
1, has shown evidence of a camera placed above the object which has been considered as space- 
to-training). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., with the teaching of McGee et al., 
because this modification would have introduced frame into frame into Matusik's et al. teaching, 
thereby "The camera is calibrated to determine internal camera geometric and optical 
characteristics (intrinsic parameters) and the 3-D position and orientation of the camera frame 
relative to the coordinate system at the station 12." (see McGee et al. col. 5, lines 54-58). 

Matusik's et al. and McGee et al. comprises.determining a respective translation 
component along three orthogonal axes, and a respective rotation component about the three 
orthogonal axes. 

Armstrong et al. (US 20040233461) teaches determining a respective translation 
component along three orthogonal axes (see sections [0011, 0030 and 0031] for three orthogonal 
axes), and a respective rotation component about the three orthogonal axes (see abs., for rotation; 
and section [0011] for three orthogonal axes). 
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It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al. and McGee et al, with the teaching 
of Armstrong's et al. because this modification would have introduced translation and rotation 
into Matusik's et al. and McGee's et al. teaching, thereby "camera coordinate systems may be 
rotated in three-dimensional space with respect to the other" (see Armstrong's et al. section 
[0026]). 

15. Claims 43 are rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik et al. 
in view of McGee et al. as applied to claim 33 above, and further in view of Horn (US 
20010034481). 

A per claim 43, Matusik et al. teaches a method that further comprising: capturing an 
image of a teaching object of a type of object that will be manipulated by the robot (see fig. 2, 
element 100); selecting a number of features from the captured image of the teaching object (see 
col. 3, lines 48-58, wherein retrieving has been taken as selecting). Matusik et al. does not 
specifically teach determining a set of object space coordinates for each of the selected features 
from the captured image of the teaching object. 

Horn (US 20010034481) teaches photogrammetric sensor position estimation including 
determining a set of object space coordinates for each of the selected features from the captured 
image of the teaching object (see section [0017], wherein a set of object.. .for each camera met 
the set of ...). 
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It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., and McGee et al. with the teaching 
of Horn, because this modification would have introduced object space coordinates into 
Matusik' s et al. and McGee et al. teaching, thereby "the camera position and field of view are 
also preferably selected so that the number of features, m, in each image are the same, and that 
the same m objects have been imaged in each image." (see Horn's section [0055]). 

16. Claims 44 are rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik et al. 
in view of McGee et al. and Horn as applied to claim 33 above, and further in view of Nayar et 
al. (US 5,802,201). 

A per claim 44, Matusik et al. teaches essential features of the invention substantially as 
claimed, but fails to teach a system wherein selecting a number of features from the captured 
image of the teaching object comprises selecting six features from the captured image of the 
teaching object. 

Nayar et al. (US 5,802,201) teaches selecting a number of features from the captured 
image (see col. 3, lines 48-58, wherein retrieving has been taken as selecting) of the teaching 
object comprises selecting six features from the captured image of the teaching object (see col. 9, 
lines 19-21, wherein the six points have been considered as six features). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al. McGee et al, and Horn with the 
teaching of Nayar et al., because this modification would have introduced six points as six 
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features into Matusik's et al. McGee's et al. and Horn's teaching, thereby "three dimensional 
information is retrieved" (see Nayar's et al. col. 4, lines 32-33). 

17. Claims 45-47 are rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik et 
al. in view of McGee et al. and Horn as applied to claim 33 above, and further in view of 
Maguire, Jr. (US 20100231706). 

A per claim 45, Matusik et al. McGee et al. and Horn teach essential features of the 
invention substantially as claimed, but they fail to teach a determining an object space-to-camera 
space transformation defining a transformation between an object space reference frame and the 
camera space reference frame. 

Maguire, Jr. (US 20100231706) teaches a camera including determining an object space- 
to-camera space transformation defining a transformation between an object space reference 
frame and the camera space reference frame (see abs.). 

It would have been obvious to a person of ordinaiy skill in the art at the time the 
invention was made to modify the teaching of Matusik et al. McGee et al. and Horn, with the 
teaching of Maguire, Jr., because this modification would have introduced reference frame of an 
object into Matusik's et al. McGee's et al. and Horn's teaching, thereby "successive images are 
computer generated images produced by a computer workstation from successive images taken 
from a camera navigated by a person within a three-axis reference frame of an object space" (see 
Mauire's Jr., claim 11). 
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A per claims 46 and 47, Matusik et al. McGee et al., Horn and Maguire, Jr. teach 
essential features of the invention substantially as claimed, but they fail to teach determining a 
position and an orientation of an object frame in the tool frame reference frame based at least in 
part on the object frame-to-camera space and camera space-to-tool space transformations; 
providing the position and orientation of the object frame to the robot; and training an intended 
operation path inside the object frame. 

Maguire, Jr. teaches determining a position and an orientation of an object frame in the 
tool frame reference frame (see section [0008], wherein the virtual reality has been considered as 
reference frame for providing images in a display) based at least in part on the object frame-to- 
camera space and camera space-to-tool space transformations(see abs., and section [0161]); 
providing the position and orientation of the object frame to the robot (see sections [0016 and 
0018]); and training an intended operation path inside the object frame (see abs.). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al. McGee et al. Horn and Maguire, Jr., 
with the teaching of Maguire, Jr., because this modification would have introduced reference 
frame of an object into Matusik's et al. McGee's et al. Horn's and Maguire's, Jr. teaching, 
thereby "successive images are computer generated images produced by a computer workstation 
from successive images taken from a camera navigated by a person within a three-axis reference 
frame of an object space" (see Mauire's Jr., claim 11). 



Application/Control Number: 10/634,874 Page 16 

Art Unit: 3664 

18. Claim 50 is rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik et al., in 
view of Horn, Bancroft et al. and Ishiyama. 



A per claim 50, Matusik et al. teaches substantially a method comprising: an algorithm 
that employs a known or determinable physical relationship between at least some of the located 
features (see fig. 2, wherein physical relationship exists between element 130 and the object 
150). Matusik et al. teaches does not teach determining an object space-to-camera space 
transformation for the target object based at least in part on a position of at least some of the 
located features using only the captured image; a single camera mounted to a movable portion of 
a robot; capturing a two-dimensional image of a volume containing a target object; locating a 
number of features in the captured image of the target object; a single camera mounted to a 
movable portion of a robot; useful in three-dimensional pose estimation for use with, the method. 

Horn (US 20010034481) teaches determining an object space-to-camera space 
transformation for the target object based at least in part on a position of at least some of the 
located features using only the captured image (see abs.). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., with the teaching of Horn, because 
this modification would have introduced object space coordinates into Matusik's et al. teaching, 
thereby "the camera position and field of view are also preferably selected so that the number of 
features, m, in each image are the same, and that the same m objects have been imaged in each 
image." (see Horn's section [0055]). 
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Matusik et al. and Horn teach essential features of the invention substantially as claimed, 
but they fail to teach a single camera mounted to a movable portion of a robot; a system that is 
useful in three-dimensional pose estimation. 

Bancroft et al. (US 6,584,375) teaches a single camera mounted to a movable portion of a 
robot (see col. 22, lines 35-38, wherein the camera has been considered mounted on a movable 
part). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., and Horn with the teaching of 
Bancroft et al., because this modification would have introduced a camera mounted to the robot 
into Matusik' s et al. and Horn's teaching, thereby " the robot equipped with audio and video 
equipment that can provide a record of what the robot observes while monitoring the retail 
environment" (see Bancroft's col. 35, lines 35-38). 

Matusik et al. Horn and Bancroft et al. teach essential features of the invention 
substantially as claimed, but they fail to teach a system that is useful in three-dimensional pose 
estimation. 

Ishiyama (US 20010033685) teaches a system that is useful in three-dimensional pose 
estimation (see abs., and section [0001]). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., Horn and Bancroft et al. with the 
teaching of Ishiyama, because this modification would have introduced frame into frame into 
Matusik's et al. Horn's and Bancroft's et al. teaching, thereby "2-D image of target object is 
obtained by use of a camera 55 as in input image." (see Ishiyama's section [001 1]). 
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19. Claims 51 and 52 are rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik 
et al., in view of Horn, Bancroft et al. and Ishiyama as applied to claim 50 above, and further in 
view of Hudson (US 20010055069). 

A per claims 51 and 52, Matusik et al., Horn and Bancroft et al. and Ishiyama teach 
essential features of the invention substantially as claimed, but they fail to teach determining at 
least one movement of the robot that orients the camera orthogonally with respect to the target 
object. 

Hudson (US 20010055069) teaches a camera orthogonally-mounted including 
determining at least one movement of the robot that orients the camera orthogonally with respect 
to the target object (see abs. and section [0003]). 

It would have been obvious to a person of ordinaiy skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., Horn Bancroft et al. and Ishiyama 
with the teaching of Hudson, because this modification would have introduced frame into frame 
into Matusik's et al. Horn's Bancroft's et al. and Ishima's teaching, thereby "2-D image of target 
object is obtained by use of a camera 55 as in input image." (see Ishiyama' s section [001 1]). 

20. Claims 53 and 58 are rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik 
et al., in view of McGee et al. and Ishiyama. 
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A per claims 53 and 58, Matusik et al. teaches a system and an associated method a 
single camera operable to capture a number of images (see abs. particularly "A system digitizes a 
three-dimensional object as a three-dimension model by ") of a calibration object means for 
calibrating the camera, by: determining a set of intrinsic parameters of the camera (see sections 
[0027-0028]) from at least one of the number of images (see abs. particularly "A system digitizes 
a three-dimensional object as a three-dimension model by ") of the calibration object captured by 
the camera (see section [0026]); and determining a set of extrinsic parameters of the camera from 
at least one of the number of images (see abs. particularly "A system digitizes a three- 
dimensional object as a three-dimension model by ") of the calibration object captured by the 
camera (see section [0026]), the set of extrinsic parameters comprising (see sections [0027 and 
0028]) determining an object space-to-camera space transformation based at least in part on a 
position of at least some of the located features in solely the captured image (see abs. particularly 
"A system digitizes a three-dimensional object as a three- dimension model"). Matusik et al. 
does not specifically teach a system useful in robotics (see fig. 2. element 100) a camera space- 
to-training space transformation defining a transformation between a camera space reference 
frame and a training space reference frame (see abs.); and locating at least six features in the 
captured image of the target object; and using an algorithm that employs a known or 
determinable physical relationship between at least some of the located features. 

McGee et al. (4,942,539) teaches useful in robotics (see fig. 2, element 100) a camera 
space-to-training space transformation defining a transformation between a camera space 
reference frame and a training space reference frame (see abs., wherein reference data has been 
reference frame; and "The offset is then transformed into the coordinate system or frame" has 
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been taken defined transformation; and fig. 1, has shown evidence of a camera placed above the 
object which has been considered as space-to-training). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., with the teaching of McGee et al, 
because this modification would have introduced frame into frame into Matusik's et al. teaching, 
thereby "The camera is calibrated to determine internal camera geometric and optical 
characteristics (intrinsic parameters) and the 3-D position and orientation of the camera frame 
relative to the coordinate system at the station 12." (see McGee's et al. col. 5, lines 54-58). 

Matusik et al., and McGee et al. teach essential features of the invention substantially as 
claimed, but fail to teach means for estimating a pose of a target object, by: capturing a two- 
dimensional image of a volume containing a target object. 

Ishiyama teaches means for estimating a pose of a target object (see abs., section [0001]), 
by: capturing a two-dimensional image of a volume containing a target object (see section 
[0003]). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., and McGee et al. with the teaching 
of Ishiyama, because this modification would have introduced frame into frame into Matusik's et 
al. and McGee's et al. teaching, thereby "2-D image of target object is obtained by use of a 
camera 55 as in input image." (see Ishiyama's section [001 1]). 



21. Claim 54 is rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik et al, 
McGee et al., and Ishiyama and Horn (US 20010034481).. 
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A per claim 54, Matusik et al. teaches a further comprising: means for training, 
comprising: capturing an image of a teaching object of a type of object that will be manipulated 
by the robot (see fig. 2, element 100); selecting a number of features from the captured image of 
the teaching object (see col. 3, lines 48-58, wherein retrieving has been taken as selecting). 
Matusik et al. does not specifically teach determining a set of object space coordinates for each 
of the selected features from the captured image of the teaching object; and determining an 
object space-to-camera space transformation defining a transformation between an object space 
reference frame and the camera space reference frame. 

Horn (US 20010034481) teaches a set of object space coordinates for each of the selected 
features from the captured image of the teaching object (see section [0017], wherein a set of 
object. ..for each camera meet the set of ...); and determining an object space-to-camera space 
transformation defining a transformation between an object space reference frame and the 
camera space reference frame (see abs.). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., and McGee et al. with the teaching 
of Horn, because this modification would have introduced object space coordinates into 
Matusik' s et al. and McGee et al. teaching, thereby "the camera position and field of view are 
also preferably selected so that the number of features, m, in each image are the same, and that 
the same m objects have been imaged in each image." (see Horn's section [0055]). 
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22. Claims 55 and 59 rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik et 
al, in view of McGee et al. and Ishiyama as applied to claims 53 and 58 above, and further in 
view of Nayar et al. (US 5,802,201).. 

A per claims 55 and 59, Matusik et al. teaches essential features of the invention 
substantially as claimed, but fails to teach a system wherein the means for estimating a pose, and 
the means for training comprises at least one programmed computer. 

Nayar et al. (US 5,802,201) teaches a system wherein the means for calibrating, the 
means for estimating a pose (see fig. 1, wherein the general purpose computer 170, has been 
considered as means for estimating pose), and the means for training comprises at least one 
programmed computer (see col. 1, lines 37-47, wherein it is known for the computer 
graphics/general purpose computer to be considered as means for holding software 
codes/routines and training function). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., with the teaching of Nayar et al., 
because this modification would have introduced computer into Matusik' s et al. teaching, 
thereby "The foregoing operations may be performed by computer 170 executing a simple 
software routine." (see Nayar's et al. col. 10, lines 21-23). 

A per claims 55 and 59, Matusik et al. teaches essential features of the invention 
substantially as claimed, but fails to teach a system wherein the means for estimating a pose, and 
the means for training comprises at least one programmed computer. 
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Nayar et al. (US 5,802,201) teaches a system wherein the means for calibrating, the 
means for estimating a pose (see fig. 1, wherein the general purpose computer 170, has been 
considered as means for estimating pose), and the means for training comprises at least one 
programmed computer (see col. 1, lines 37-47, wherein it is known for the computer 
graphics/general purpose computer to be considered as means for holding software 
codes/routines and training function). 

It would have been obvious to a person of ordinaiy skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., with the teaching of Nayar et al, 
because this modification would have introduced computer into Matusik's et al. teaching, 
thereby "The foregoing operations may be performed by computer 170 executing a simple 
software routine." (see Nayar's et al. col. 10, lines 21-23). 

23. Claims 56 and 60 are rejected under 35 U.S.C. 103(a) as being unpatentable over Matusik 
et al., in view of McGee et al. and Ishiyama as applied to claims 53 and 58 above, and further in 
view of Nayar et al. 

A per claims 56 and 60, Matusik et al. and McGee et al. teach essential features of the 
invention substantially as claimed, but they fail to teach means for calibrating and the means for 
estimating a pose comprises at least one computer-readable medium storing instructions 
operating at least one computer. 

Nayar et al. teaches means for calibrating and the means for estimating a pose (see col. 1, 
lines 37-47, wherein the computer graphics has been considered as means for holding software 
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codes and training) comprises at least one computer-readable medium storing instructions 
operating at least one computer (see col. 3, lines 27-32). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify the teaching of Matusik et al., with the teaching of Nayar et al, 
because this modification would have introduced computer into Matusik' s et al. teaching, 
thereby "The foregoing operations may be performed by computer 170 executing a simple 
software routine." (see Nayar's et al. col. 10, lines 21-23). 

Allowable Subject Matter 

24. Claim 49, 57 and 61 are objected to as being dependent upon a rejected base claim, but 
would be allowable if rewritten in independent form including all of the limitations of the base 
claim and any intervening claims. 

25. The following is a statement of reasons for the indication of allowable subject matter: 
With respect claims 49, 57 and 61, the prior art of record fail to teach or fairly suggest 

adjusting a position of the movable portion of the robot if the number of features located in the 
captured image of the target object is determined to be an insufficient number of features. 
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26. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to MCDIEUNEL MARC whose telephone number is (571)272- 
6964. The examiner can normally be reached on 6:30-5:00 Mon-Thu. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Khoi Tran can be reached on (571) 272-6919. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 

/McDieunel Marc oV 
/KHOI TRAN/ 

Supervisory Patent Examiner, Art Unit 3664 



